{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(advanced-data)=\n",
    "# Advanced Data\n",
    "\n",
    "```{note}\n",
    "If you're just starting to code, you can safely skip this chapter.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this chapter, you'll learn about advanced methods for working with data.\n",
    "\n",
    "### Imports\n",
    "\n",
    "First we need to import the classic packages we almost always need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set max rows displayed for readability\n",
    "pd.set_option(\"display.max_rows\", 6)\n",
    "# Plot settings\n",
    "plt.style.use(\n",
    "    \"https://github.com/aeturrell/coding-for-economists/raw/main/plot_style.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Validation\n",
    "\n",
    "When working with data, it's useful to have a way to validate that what you're getting is what you *think* you're getting. And so various data validation methods have appeared that will fail loudly and proudly when your data are not as expected. Although most useful in a situation where data are being regularly updated, for example when producing a monthly index, these tools can also be useful if you want to be extra careful with your data in a research paper, if you are scaling up from a sample to a bigger dataset, or if you are updating an existing project.\n",
    "\n",
    "Of course, the most minimal way to do some validation is to ensure that your dataframe has its columns cast as the right data types from the start, and this is good practice in general. That's about *data type*, but these tools go beyond that and deal with values *within* a data type. So rather than just saying it's an integer, a data validation library might say that a column must be an integer between 0 and 100.\n",
    "\n",
    "The normal way that data validation libraries are used is that you specify some expectations you have of the dataset that will be flagged if and when they fail. Let's see some of the tools that are available that do this. As ever, this chapter has benefitted enormously from the documentation of the featured packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandera\n",
    "\n",
    "[**Pandera**](https://pandera.readthedocs.io/en/stable/index.html) {cite}`niels_bantilan-proc-scipy-2020` has tight integration with **pandas** (as the name suggests). There are several different ways to specify data validation checks.\n",
    "\n",
    "#### Dataframe style checks\n",
    "\n",
    "It's easiest to show how it works with an example. We'll first create some data that we'd like to validate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandera as pa\n",
    "\n",
    "# data to validate\n",
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"column1\": [1, 4, 0, 10, 9],\n",
    "        \"column2\": [-1.3, -1.4, -2.9, -10.1, -20.4],\n",
    "        \"column3\": [\"value_1\", \"value_2\", \"value_3\", \"value_2\", \"value_1\"],\n",
    "    }\n",
    ")\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**pandera** (I'm going to spell it as pantera sooner or later) works by defining *data schemas* for each column that say what values should appear in that column. Let's create a schema for this dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define schema\n",
    "schema = pa.DataFrameSchema(\n",
    "    {\n",
    "        \"column1\": pa.Column(int, checks=pa.Check.le(10)),\n",
    "        \"column2\": pa.Column(float, checks=pa.Check.lt(-1.2)),\n",
    "        \"column3\": pa.Column(\n",
    "            str,\n",
    "            checks=[\n",
    "                pa.Check.str_startswith(\"value_\"),\n",
    "                # define custom checks as functions that take a series as input and\n",
    "                # outputs a boolean or boolean Series\n",
    "                pa.Check(lambda s: s.str.split(\"_\", expand=True).shape[1] == 2),\n",
    "            ],\n",
    "        ),\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What did we ask for here? That:\n",
    "\n",
    "- column 1 should be an integer that is less than or equal to 10\n",
    "- column 2 should be a continuous variable that's less than 1.2\n",
    "- column 3 should be a string that begins 'value_' and has a custom check on it that it can be split into chunks based on the single appearance of an underscore.\n",
    "\n",
    "Alright, let's see what happens when we put well-behaved data through the schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validated_df = schema(df)\n",
    "print(validated_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data comes through the schema without incident because the data satisfies all the checks. Now, what about some data that *aren't* well-behaved, but in a single column (here 1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# data to validate\n",
    "bad_df = pd.DataFrame({\n",
    "    \"column1\": [11, 4],\n",
    "    \"column2\": [-1.3, -1.4],\n",
    "    \"column3\": [\"value_1\", \"value_2\"],\n",
    "})\n",
    "schema(bad_df)\n",
    "```\n",
    "\n",
    "```\n",
    "SchemaError                               Traceback (most recent call last)\n",
    "<ipython-input-6-ff71778da6fd> in <module>\n",
    "      5     \"column3\": [\"value_1\", \"value_2\"],\n",
    "      6 })\n",
    "----> 7 schema(bad_df)\n",
    "...\n",
    "...\n",
    "SchemaError: <Schema Column: 'column1' type=<class 'int'>> failed element-wise validator 0:\n",
    "<Check less_than_or_equal_to: less_than_or_equal_to(10)>\n",
    "failure cases:\n",
    "   index  failure_case\n",
    "0      0            11\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error message is rather verbose but scroll to the end and we see the cause of it: a 'schema error'. Not only that but we get a report about what failed (column 1), what check it was (0), and even what value triggered the error (11).\n",
    "\n",
    "Use `schema.validate(df, lazy=True)` to get all of the errors rather than just the first one to be triggered.\n",
    "\n",
    "You can also pass hypothesis tests into a schema as a check; there's a nice [example of using a two-sample t-test](https://pandera.readthedocs.io/en/stable/hypothesis.html) in the documentation.\n",
    "\n",
    "#### Data checks as a class\n",
    "\n",
    "The 'dataframe' style of checks is not the only way to specify a schema. If you're au fait with classes, you can also specify checks that way. Here's the same example in that form:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pandera.typing import Series\n",
    "\n",
    "\n",
    "class Schema(pa.SchemaModel):\n",
    "\n",
    "    column1: Series[int] = pa.Field(le=10)\n",
    "    column2: Series[float] = pa.Field(lt=-1.2)\n",
    "    column3: Series[str] = pa.Field(str_startswith=\"value_\")\n",
    "\n",
    "    @pa.check(\"column3\")\n",
    "    def column_3_check(cls, series: Series[str]) -> Series[bool]:\n",
    "        \"\"\"Check that column3 values have two elements after being split with '_'\"\"\"\n",
    "        return series.str.split(\"_\", expand=True).shape[1] == 2\n",
    "\n",
    "\n",
    "Schema.validate(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, this passes but again it would not have with `bad_df` as defined earlier.\n",
    "\n",
    "It doesn't *have* to be a single column that you're checking at a time, **pandera** can also check data that is the subject of `groupby` operations. Here's an example where data are grouped by a `group` column and then the means of the `height` column are compared:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"height\": [5.6, 6.4, 4.0, 7.1],\n",
    "        \"group\": [\"A\", \"B\", \"A\", \"B\"],\n",
    "    }\n",
    ")\n",
    "\n",
    "schema = pa.DataFrameSchema(\n",
    "    {\n",
    "        \"height\": pa.Column(\n",
    "            pa.Float, pa.Check(lambda g: g[\"A\"].mean() < g[\"B\"].mean(), groupby=\"group\")\n",
    "        ),\n",
    "        \"group\": pa.Column(pa.String),\n",
    "    }\n",
    ")\n",
    "\n",
    "schema.validate(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data checks for pipelines\n",
    "\n",
    "You can also specify the validations as function decorators that check the data as it *enters* (`check_input`) or *exits* (`check_output`) a function (or both, with `check_io`). This could be really useful as part of a larger pipeline of data processes. Here's an example (of a dataframe that passes the input checks):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"column1\": [1, 4, 0, 10, 9],\n",
    "        \"column2\": [-1.3, -1.4, -2.9, -10.1, -20.4],\n",
    "    }\n",
    ")\n",
    "\n",
    "in_schema = pa.DataFrameSchema(\n",
    "    {\n",
    "        \"column1\": pa.Column(\n",
    "            pa.Int, pa.Check(lambda x: 0 <= x <= 10, element_wise=True)\n",
    "        ),\n",
    "        \"column2\": pa.Column(pa.Float, pa.Check(lambda x: x < -1.2)),\n",
    "    }\n",
    ")\n",
    "\n",
    "\n",
    "@pa.check_input(in_schema)\n",
    "def preprocessor(dataframe):\n",
    "    dataframe[\"column3\"] = dataframe[\"column1\"] + dataframe[\"column2\"]\n",
    "    return dataframe\n",
    "\n",
    "\n",
    "preprocessed_df = preprocessor(df)\n",
    "preprocessed_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Great Expectations\n",
    "\n",
    "[**Great Expectations**](https://greatexpectations.io/) is another data validation library. It's really geared toward to production (and can integrate well with some other production tools). I wouldn't necessarily recommend it if you're doing a research project but it's got some amazing features that just mean it made it into this chapter--namely that it doesn't just validate data, but it documents and profiles it too.\n",
    "\n",
    "Their philosophy is that 'Expectations', the checks or assertions about data, are unit tests for your data (don't worry if you're not familiar with unit tests-they are checks for your code!). Once you have run your expectations, the library can create data documentation and data quality reports from them. The data documentation is quite an amazing feature, providing a navigable HTML report of your dataset.\n",
    "\n",
    "Using **Great Expectations** is a bit different from **pandera** as it replaces your dataframe with a **Great Expectations** `PandasDataset` that looks and feels just like a regular pandas dataframe but has extra methods related to data validation. To convert a regular pandas dataframe, you wrap it with `ge.from_pandas`. \n",
    "\n",
    "Let's look at some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import great_expectations as ge\n",
    "\n",
    "df = ge.from_pandas(\n",
    "    pd.DataFrame(\n",
    "        {\n",
    "            \"column1\": [1, 4, 0, 10, 9],\n",
    "            \"column2\": [-1.3, -1.4, -2.9, -10.1, -20.4],\n",
    "            \"column3\": [\"value_1\", \"value_2\", \"value_3\", \"value_2\", \"value_1\"],\n",
    "        }\n",
    "    )\n",
    ")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run some expectations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.expect_column_values_to_be_of_type(\"column1\", \"int\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.expect_column_proportion_of_unique_values_to_be_between(\"column3\", 3, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the first expectation passed, while the second failed but did *not* throw an error... we were just told in JSON.\n",
    "\n",
    "Let's also see a less stringent expectations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.expect_column_values_to_be_between(\n",
    "    column=\"column2\",\n",
    "    min_value=-15,\n",
    "    max_value=0,\n",
    "    mostly=0.80,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having a bunch of JSON thrown at you is hardly a pleasant experience, and running a few commands solely in a notebook (which is how this was generated) is not really the way that **Great Expectations** is intended to be used. Instead, it is designed to be used via several commands in the terminal that set up folders and files that will track the expectations you create and produce nice HTML reports on your data and whether it passed your tests or not. There is a [full tutorial](https://docs.greatexpectations.io/en/latest/guides/tutorials/getting_started.html) available on the website."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pydantic\n",
    "\n",
    "[**Pydantic**](https://pydantic-docs.helpmanual.io/) has some of the same features as **pandera** but it piggybacks on the ability of Python 3.5+ to have 'typed' variables (if you're not sure what that is, it's a way to declare a variable has a particular data type from inception) and it is really focused around the validation of objects (created from classes) rather than dataframes.\n",
    "\n",
    "Here's an example of a Pydantic schema that also implements a class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "from typing import List, Optional\n",
    "from pydantic import BaseModel\n",
    "\n",
    "\n",
    "class User(BaseModel):\n",
    "    id: int\n",
    "    name = \"Katherine Johnson\"\n",
    "    signup_ts: Optional[datetime] = None\n",
    "    friends: List[int] = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now pass some (acceptable) external data into this schema, creating an object of type `User`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "external_data = {\n",
    "    \"id\": \"123\",\n",
    "    \"signup_ts\": \"2019-06-01 12:22\",\n",
    "    \"friends\": [1, 2, \"3\"],\n",
    "}\n",
    "user = User(**external_data)\n",
    "user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That all worked well (as expected) but what if we now create an object that has 'bad' data:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "User(signup_ts='broken', friends=[1, 2, 'not number'])\n",
    "```\n",
    "\n",
    "```\n",
    "---------------------------------------------------------------------------\n",
    "ValidationError                           Traceback (most recent call last)\n",
    "<ipython-input-15-7a13cdd5b253> in <module>\n",
    "----> 1 User(signup_ts='broken', friends=[1, 2, 'not number'])\n",
    "\n",
    "~/opt/anaconda3/envs/codeforecon/lib/python3.8/site-packages/pydantic/main.cpython-38-darwin.so in pydantic.main.BaseModel.__init__()\n",
    "\n",
    "ValidationError: 3 validation errors for User\n",
    "id\n",
    "  field required (type=value_error.missing)\n",
    "signup_ts\n",
    "  invalid datetime format (type=value_error.datetime)\n",
    "friends -> 2\n",
    "  value is not a valid integer (type=type_error.integer)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a loud failure, a `ValidationError`, and a short report about what went wrong."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cerberus\n",
    "\n",
    "Our final stop on this tour of data validation is [**Cerberus**](https://docs.python-cerberus.org/en/stable/). It is designed to be a simple and lightweight data validation functionality. In this case, the schema is specified as a JSON file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "schema = {\"name\": {\"type\": \"string\"}, \"score\": {\"type\": \"integer\", \"max\": 10}}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we pass data in to a **Cerberus** validator object along with the schema. This will return true for good data but for bad data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cerberus import Validator\n",
    "\n",
    "v = Validator()\n",
    "data = {\"name\": \"Sadie Alexander\", \"score\": 12, \"occupation\": \"Economist\"}\n",
    "v.validate(data, schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is useful to know but we might want a bit more detail:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.errors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much more complex schemas can be [constructed](https://docs.python-cerberus.org/en/stable/validation-rules.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Orchestration\n",
    "\n",
    "Data orchestration is the automation of data-driven processes from end-to-end, including preparing data, making decisions based on that data, and taking actions based on those decisions. Think of a data pipeline in which you extract data (perhaps from files), transform it somehow, and load it into where you want to put it (downstream in your research process, or perhaps into an app). \n",
    "\n",
    "There are some truly amazing tools out there to help you do this on a production scale. Perhaps the best known and most suited for production are AirBnB's [**Airflow**](http://airflow.incubator.apache.org/) and Spotify's [**Luigi**](https://github.com/spotify/luigi). Airflow in particular is widely used in the tech industry, and doesn't just schedule data processes in Python: it can run processes in pretty much whatever you like. Both of these tools try to solve the 'plumbing' associated with long-running batch processes on data: chaining tasks, automating them, dealing with failures, and scheduling. Both Luigi and Airflow have fancy interfaces to show you what's going on with your tasks. \n",
    "\n",
    "Data orchestration tools typically have a directed acyclic graph, a DAG, at the heart of how tasks are strung together. This defines how different tasks depend on each other in order, with one task following from the previous one (or perhaps following from multiple previous tasks). It's an automation dream.\n",
    "\n",
    "However, for a research project, it's hard to recommend these two tools as they're just a bit too powerful; Airflow in particular can do just about anything but has a steep learning curve. So, instead, to show the power of data orchestration we'll use a more lightweight but also very powerful library: [**dagster**](https://dagster.io/), which bills itself as 'a data orchestrator for machine learning, analytics, and ETL [extract, transform, and load]'. Some of the key features are being able to implement components in various tools, such as Pandas and SQL, define pipelines as DAGs, and to test the same setup on your machine that you then deploy to cloud. Like the other tools, it has a nice visual interface to show what's happening. Necessarily, we'll only be seeing a very brief introduction to it here.\n",
    "\n",
    "**Dagster** works by using *function decorators* (the `@` symbol) to define the 'solids' that are the nodes in **dagster**'s DAG. The `@pipeline` defines the set of functions that are composed in the pipeline. Here's an example of defining a simple pipeline that produces the same phrase with different capitalisations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dagster import execute_pipeline, pipeline, solid\n",
    "\n",
    "\n",
    "@solid\n",
    "def get_name(_):\n",
    "    return \"dagster\"\n",
    "\n",
    "\n",
    "@solid\n",
    "def hello(context, name: str):\n",
    "    return \"hello, {name}!\".format(name=name)\n",
    "\n",
    "\n",
    "@solid\n",
    "def leading_caps(context, phrase: str):\n",
    "    return phrase.title()\n",
    "\n",
    "\n",
    "@solid\n",
    "def all_upper(context, phrase: str):\n",
    "    return phrase.upper()\n",
    "\n",
    "\n",
    "@solid\n",
    "def display(context, phrase_to_display: str):\n",
    "    print(phrase_to_display)\n",
    "\n",
    "\n",
    "@pipeline\n",
    "def hello_pipeline():\n",
    "    hello_text = hello(get_name())\n",
    "    # Alias distinguishes between the same fn called twice with different params\n",
    "    display_leading = display.alias(\"leading\")\n",
    "    display_upper = display.alias(\"upper\")\n",
    "    display_leading(leading_caps(hello_text))\n",
    "    display_upper(all_upper(hello_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is purposefully both over-engineered and very simple (Dagster pipelines can have a *lot* more complexity of this), but we're just seeing the most basic principles of the library here.\n",
    "\n",
    "With a pipeline defined, we can now execute it in three different ways: in a python script, using the command line, or via graphical user interface. Let's see the script method first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [],
   "source": [
    "if __name__ == \"__main__\":\n",
    "    execute_pipeline(hello_pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a lot of output about how the pipeline executed but it ends with `HELLO, DAGSTER!` and `Hello, Dagster!`, which is the result of our DAG.\n",
    "\n",
    "The second way of running it is by using the command line. If this was in a script called `hello_dagster.py` then the command to run it would be:\n",
    "\n",
    "```\n",
    "dagster pipeline execute -f hello_dagster.py\n",
    "```\n",
    "\n",
    "Finally, there's a nice GUI (graphical user interface) that you can use to run jobs too. This is launched from the command line via `dagit -f hello_dagster.py`. Once you have opened the link using a browser, navigate to 'Playground' and use the 'Launch Execution' button to set off a run. When you open the GUI, you'll see the DAG set out for you like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO update\n",
    "from IPython.display import Image\n",
    "\n",
    "Image(os.path.join(\"img\", \"data_dag_gui.png\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a lot of benefits to thinking about your data processing in terms of pipelines and DAGs. You can see the last time any particular node executed and, if node execution fails, you can see why, where, and how. The GUI has tons of extra functionality too; for instance, you can track what’s produced by your pipelines with the 'Asset Manager', so you can understand how your data was generated and trace issues when it doesn’t look how you expect. There's also optional typing of data types on input and output which helps you to catch bugs in data like you'd catch bugs in code (in the above, we declared all of the input types to be strings but didn't declare what the output types were so they appeared as 'any').\n",
    "\n",
    "This has barely scratched the surface, but hopefully it's given you an insight into what data orchestration tools do and how they might be helpful for complex data workflows."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "interpreter": {
   "hash": "671f4d32165728098ed6607f79d86bfe6b725b450a30021a55936f1af379a247"
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 64-bit ('codeforecon': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
